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Major  Weapon  System  Portfolio  Analysis 


•  We  assessed  cost  and  schedule  performance  of 
DOD’s  portfolio  of  95  Major  Weapon  System 
programs 

•  Data  was  obtained  from  Selected  Acquisition 
Reports 

•  Missile  Defense  Agency  reports  only  one 
SAR  for  its  systems 
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DOD  Has  Increased  Its  Commitment  In  Major 
Defense  Acquisitions  Programs . 


Fiscal  Year  2008  Dollars  in  Billions 


FY  2000  Porfolio  FY  2005  Portfolio  FY  2007  Portfolio 


□  Outstanding  M  Expended 

Source:  GAO  analysis  of  DOD  data. 
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But  DOD  Outcomes  Are  Not  Improving 


50%  n 


Change  in  Total  Change  in  RDT&E  Share  of  programs  with 

Acquisition  Costs  From  Costs  From  First  25  percent  cost  growth 

First  Estimates  Estimates 


□  FY  2000  Portfolio  □  FY  2005  Portfolio  ■  FY  2007  Portfolio 


Months 


Source:  GAO  analysis  of  DOD  data. 


■  FY  2000  Portfolio  □  FY  2005  Portfolio  □  FY  2007  Portfolio 
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2008  Weapon  System  Assessment 


•  We  assessed  cost  and 
schedule  performance  of 
72  DOD  programs 


la  }  fSr  li©  Origina  I'Artist' 
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Little  Evidence  of  Widespread  Adoption  of 
Knowledge-based  Acquisition  Process 


•  DOD’s  acquisition  practices  necessary  to  ensure  effective  implementation  of 

knowledge-based  process  are  not  always  followed  despite  policies  and  guidance  to 
the  contrary. 


Key 

junctures  Development  start  Design  review  Production  start 


Knowledge  point  1 


Knowledge  point  2 


Knowledge  point  3 


Best  Mature  all  critical 

practices  technologies 


Achieve  knowledge  point 
1  on  time  and  complete 
90  percent  of  engineering 
drawings 


Achieve  knowledge  points 
1  and  2  on  time,  and  have  all 
critical  processes  under 
statistical  control 


DOD 

outcomes3 


12  percent 
of  programs 


4  percent  of 
programs 


0  percent  of 
programs13 


Source:  GAO  presentation  of  DOD  data. 

a  Not  all  programs  provided  information  for  each  knowledge  point  or  had  passed  through  all  three  key  junctures, 
b  In  our  assessment  of  two  programs,  the  Light  Utility  Helicopter  and  the  Joint  Cargo  Aircraft,  are  depicted  as  meeting  all  three 
knowledge  points  when  they  began  at  production  start.  We  excluded  these  two  programs  from  our  analysis  because  they  were 
based  on  commercially  available  products  and  we  did  not  assess  their  knowledge  attainment  with  our  best  practices  metrics. 
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Programs  Enter  System  Development  Without 
Mature  Technologies 


Percent  of  Programs  Achieving  Technology 
Maturity  At  Key  Junctures 


| _ |  2005  assessment 

|  |  2006  assessment 

Source:  GAO  analysis  of  DOD  data. 


2007  assessment 

2008  assessment 


46  percent  of  technologies 
(164  out  of  356)  in  immature 
state. 


Cost  growth  for  programs  with 
immature  technologies  was  44 
percent  higher. 


Less  than  10  percent  of 
programs  had  completed 
preliminary  design  review 
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Programs  Continue  to  Move  Into  System 
Demonstration  Without  Achieving  Design  Stability 


Percent  of  Programs  Releasing  90  Percent  of 
Engineering  Drawing  At  Key  Junctures 

Percent 


|  |  2005  assessment 

|  |  2006  assessment 

Source:  GAO  analysis  of  DOD  data. 


2007  assessment 

2008  assessment 


Over  half  of  programs  did  not 
have  mature  technologies  by 
design  review. 


More  than  75  percent  of 
programs  did  not  meet 
drawings  release  best 
practices. 


Over  80  percent  of  programs 
did  not  demonstrate  system 
integration  through  integration 
lab  or  prototype  by  design 
review. 
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Programs  Enter  Production  Without  Demonstrating 
Acceptable  Manufacturing  and  Test  Performance 


•  Roughly  one-third  of  the  programs  did  not  reach  either  technology 
or  design  maturity  by  production  commitment. 


•  None  of  the  programs  that  had  production  decisions  provided  data 
showing  they  had  all  their  processes  in  statistical  control. 


•  Many  programs  still  had  SDD  activities  remaining  when  entering 
production. 
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Underlying  Deficiencies  and  Opportunities  for 
Increasing  Program  Knowledge  Early 


•  Systems  Engineering  and  Business  Cases 

•  Technology  Transition 

•  Portfolio  Management 

•  Program  Management  Turnover 

•  Cost  Estimates  and  Full  Funding 
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Systems  Engineering  Provides  Evidence  that 
Product  Can  Be  Developed  Within  Resources 


•  Business  case  should  provide  evidence: 

(1 )  Warfighter  needs  are  valid  and  can  be  met  with  chosen  concept,  and 

(2)  The  chosen  concept  can  be  developed  and  produced  within  resources- 
technologies,  funding,  design  knowledge,  and  time. 


•  Early  systems  engineering  enables  a  developer  to  identify  and  resolve 
gaps  between  resources  and  requirements  before  product  development 
begins. 
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Solid,  Executable  Business  Case  Needed  For 
Programs 


•  DOD  often  sets  optimistic  requirements  that  requires  new  and 
unproven  technologies  that  cannot  be  met  within  available 
resources. 

•  While  DOD’s  acquisition  policy  is  informed  with  systems 
engineering  rules,  the  absence  of  disciplined  and  timely  practices 
leads  to  uninformed  requirements. 


•  When  early  requirements  analysis  is  not  adequately  performed  to 
ensure  DOD  needs  can  be  met  within  resources,  increased  costs 
risk  to  government  can  occur. 


•  Many  programs  have  reset  their  business  case  at  least  once. 
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Stronger  Practices  Needed  To  Improve  DOD’s 
Technology  Transition  (GAO-06-883) 


•  Lacks  the  strong  influence  at  the  corporate  level  to  guide  the 
department’s  technology  investments. 

•  DOD  does  not  use  gated  processes  with  criteria  that  allows  lab  and 
program  managers  to  know  when  technologies  are  ready  for 
transition. 

•  Funding  is  not  aligned  to  effectively  and  efficiently  transition 
technologies  to  programs. 


•  Technologies  are  not  mature  when  needed,  forcing  programs  to  pull 
them  too  early. 
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Integrated  Portfolio  Management  Approach 
Needed  For  Weapons  Investment  (GA0-07-388) 


•  DOD  largely  continues  to  define  warfighter  needs  and 
make  investment  decisions  in  a  service-centric  way. 


•  DOD  also  assesses  warfighting  needs  and  their  funding 
implications  under  separate  decision-making  processes. 


•  DOD’s  approach  impedes  ability  to  prioritize  needs  so 
that  it  pursues  not  only  the  ones  most  important  but  also 
ones  it  can  afford. 
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Additional  Factors  Influence  DOD’s  Ability  to 
Manage  Programs  and  Improve  Outcomes 


Source:  GAO. 


PRESSURE  ON 
DECISION  MAKER  TO  ... 

...  promise  high 
performance 


...  promise  low 
resource  demands 


...  move  forward, 
get  knowledge  later 
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Frequent  Changes  to  Program  Management 
Reduces  Accountability 


•  Frequent  turnover  makes  it  difficult  to  hold  one  program 
manager  accountable  for  established  business  case. 


•  Program  managers  have  little  incentive  to  pursue  knowledge- 
based  approach  as  program  funding  is  not  tied  to  achieving 
knowledge. 


•  Lengthy  program  cycle  times  make  program  management 
longevity  difficult. 
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Realistic  Cost  Estimates  and  Full  Funding 
Needed  (GAO-08-619) 

•  Program  budget  is  established  well  before  program  begins. 

•  5  or  6  year  FYDP  does  not  capture  the  “full”  costs  of  programs. 

•  Cost  estimates  are  optimistic  and  based  on  limited  knowledge. 

•  Most  estimates  (“most  probable  cost”)  do  not  provide  visibility 
into  uncertainty  and/or  risk. 

•  Pressure  to  “fit  programs  into  the  FYDP”  /  competition  for 
funding. 
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Observations  On  Actions  Needed  From 
Related  GAO  Work 


•  Establishing  an  enterprise-wide  portfolio  management  approach; 


•  Constraining  individual  program  requirements  within  available 
resources,  leveraging  systems  engineering; 


•  Enabling  science  and  technology  organizations  to  shoulder  the 
technology  burden; 


•  Establishing  sound,  executable  business  cases; 


•  Establishing  and  enforcing  controls  to  ensure  that  appropriate 
knowledge  is  captured  and  used  at  critical  junctures; 


•  Holding  program  managers  and  decision-makers  accountable. 


